Relevant and robust: a response to Marcus and Davis (2013).

Commentary Computational models in psychology are precise, fully explicit scientific hypotheses. Probabilistic models in particular formalize hypotheses about the beliefs of agents— their knowledge and assumptions about the world—using the structured collection of probabilities referred to as priors and likelihoods. The probability calculus then describes inferences that can be drawn by combining these beliefs with new evidence, without the need to commit to a process-level explanation of how these inferences are performed (Marr, 1982). Over the past 15 years, probabilistic modeling of human cognition has yielded quantitative theories of a wide variety of phenomena (2013) critiqued several examples of this work on the basis of the existence of alternative models and potentially inconsistent data, using these critiques to question the basic validity of the probabilistic-models approach. Contra the broad rhetoric of their article, however, the points made by Marcus and Davis— although useful to consider—do not indicate systematic problems with the probabilistic-modeling enterprise. Several of Marcus and Davis's objections stem from a fundamental confusion about the status of optimality in probabilistic modeling, which has been discussed in responses to other critiques (see Frank, 2013; Griffiths, Chater, Norris, & Pouget, 2012). Briefly, an optimal analysis is not the optimal analysis for a task or domain. Different probabilistic models instantiate different psychological hypotheses. Optimality provides a bridging assumption between these hypotheses and human behavior—one that can be reexamined or overturned as the data warrant. Marcus and Davis argued that individual probabilistic models require a host of potentially problematic modeling choices. Indeed, probabilistic models are created via a series of choices concerning, among other things, priors, likelihoods, and response functions. Each of these choices embodies a proposal about cognition, and these proposals will often be wrong. Identifying model assumptions that result in a mismatch to empirical data allows these assumptions to be replaced or refined. Systematic iteration to achieve a better model is part of the normal progress of science. But if choices are made post hoc, a model can be overfit to the particulars of the empirical data. Marcus and Davis suggested that certain of our models suffer from this problem. For instance, they showed that data on pragmatic inference (Frank & Goodman, 2012) are inconsistent with an alternative variant of the proposed model that uses a hard-max rather than a soft-max function, and they asked whether our choice of a soft-max rule was dependent on the data. The soft-max rule is …